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VERSION WITH MARKINGS TO SHOW CHANGES MADE 

BACKGROUND OF THE INVENTION 

This application claims priority to French application number 02016733 filed 
December 19, 2002. 

This invention relates to a method of discretization / grouping of a source attribute or 
a group of source attributes of a database containing a population of individuals with the 
object in particular of predicting modalities of a given target attribute. The invention 
particularly finds application in the statistical handling of data, in particular in the domain of 

supervised learning. 

The statistical analysis of data (also called "data mining") has gained considerable 
ground in recent years with the extension of electronic commerce and the appearance of very 
large databases. Data mining aims in a general way to explore, classify and extract 
underlying rules of associations within a database. In particular, it is used to constmct 
classification or prediction models. The classification makes it possible to identify, within 
the database, categories fi-om combinations of attributes, and then to arrange the data as a 
function of these categories. 

In a general way, the values (also called modalities) taken by an attribute may be 
numeric (for example, a bill of sale) or symbolic (for example, a category of consumption). 
In the first case we speak of a numeric attribute and in the second case of a symbolic 
attribute. 

Some methods of data mining require a "discretization" of the numeric attributes. By 
discretization of a numeric attribute we understand here a division of the domain of values 
taken by an attribute into a finite number of intervals. If the domain in question is a range of 
continuous values the discretization is expressed by a quantification of this range. If this 
domain is already made up of discrete ordered values, discretization will have the fiinction of 
regrouping these values into groups of consecutive values. 

The discretization of numeric attributes has been widely treated in the literatvire. For 
example, a description of it is found in the work of Zighed et al. under the title "Graphes 
d'induction" ["Induction Graphs"] published by Hermes Science Publications. 

We distinguish two types of discretization methods: descending methods and 
ascending methods. The descending methods start fi-om the complete interval to be 
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discretized and seek the best cut-off point of the interval by optimizing a predetermined 
criterion. The ascending methods start from elementary intervals and seek the best merge of 
two adjacent intervals by optimizing a predeterimined criterion. In both cases, they are 
applied iteratively until a stopping criterion is satisfied. 

SUMMARY OF THE INVENTION 
A method of discretization / grouping of a source attribute or of a source attributes 
group of a database. This invention relates to a method of discretization / grouping of a 
source attribute or of a source attributes group of a database containing a population of 
individuals with the object in particular of predicting modalities of a given target attribute. 
The method includes the steps of: 

a) partitioning of the modalities of the source attribute or the attribute group into 
elementary regions, 

b) evaluating of a merge criterion for each pair of elementarv regions, 

c) searching, among the set of pairs of elementarv regions that can be merged, for the 
pair of elementarv regions for which the merge criterion would be optimized, 

d) skipping directly to step f) as long as the value of a valuation variable of the merge 
under consideration is not within a predetermined zone of atypical values, 

e) Stopping of the method if there are no elementarv regions whose merge would have 
the consequence of improving said merge criterion, and 

f) otherwise merging and reiteration of steps b) to e). 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 is a flowchart of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

This invention relates most particularly to an ascending discretization method based 
on the global optimization of the criterion. 

An ascending discretization method using the ^ criterion is known in the literature 
under the name ChiMerge. It is described, for example, in the document entitled 
"Discretization of Numeric Attributes" published in Proceedings Tenth National Conference 
on Artificial Intelligence, San Jose, CA, USA, 12-16 July 1992, pages 123 -128 under the 
name of R. Kerbe [intemet says R. Kerber]. 
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It is to be recalled in the first place that the criterion makes it possible under certain 
assumptions to determine the degree of independence of two random variables. 

Given S a source attribute and T a target attribute. We will suppose, to fix our ideas, 
that S presents five modalities a, b, c, d, e and T three modalities A, B, C. Table 1 shows the 
contingence table of the variables S and T with the following conventions: 

Viij is the number of individuals observed for the ^ modality of the variable S and the 
modality of the variable T. rtij is also called the observed effective of the cell (z,^); 

«/ is the total number of individuals for the modality of the variable S. w/. is also 
called the observed effective of the line /; 

rty is the total number of individuals for the modality of the variable T. n,j is also 
called the observed effective of the column j\ 

Nis the total number of individuals. 



S/T 


A 


B 


C 


Total 


a 


nji 


nj2 


ni3 




b 


nil 


nil 


n23 




c 


njj 


nsi 


nss 




d 


n4i 


n42 


n43 




e 


nsi 


nsi 


nss 




Total 


nj 


ni 


ns 


Article III. N 



Table 1 



Generally speaking, we note the number of modalities of the attribute S and the 
number of modalities of the attribute T as / and 7 respectively. 

We define the theoretical effective e,y of the cell (ij) by ey = ^''^'^ , representing the 

number of individuals that would be observed in the cell of the contingence table in the case 
of independent variables. The deviation fi-om independence of the variable S and T is 
measured by: 

= ±±^^1^ (1) 
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The higher the value of x^, the less probable is the assumption of independence of the 
random variables S and T. We speak with abuse of language of probability of independence 
of the variables. 

More precisely, is a random variable whose density can be shown to follow a fixed 
law of with (/-I ),(*/-!) degrees of fi-eedom. The law of is that followed by a quadratic 
sum of centered normal random values. It has, in fact, the expression of a y law and tends 
toward a guassian law when the number of degrees of fi'eedom is high. 

For example, if 7=5 and J=3, the number of degrees of fireedom has the value of 8. If 
the value of x^ calculated by (1) is 20, the law of x^ with 8 degrees of fi-eedom gives a 
probability of independence of S and T of 1%. 

Having shown that the criterion makes it possible to determine the degree of 
independence of two random variables, we will now present the ascending discretization 
method through optimization of the criterion constituted by the method referred to as 
ChiMerge. 

We consider the general case of a source attribute S with /modalities and an attribute 
T with Jmodalities. The ChiMerge method considers only two consecutive lines / and z+l of 
the contingence table. Let q 'i,q '2,,.,q 'j be the local distribution (i.e., in the local context of 
the consecutive lines / and z+1) of probabiUty of the modaUties for the target attribute T. If «/. 
is the effective of the line / and Wi+i. is the effective of the line the observed and 
theoretical effectives of the line i are expressed respectively by nij^aynt and eij=q jnt where 
the a,y represent the proportions of effectives observed for the line In the same way, the 
observed and theoretical effectives of the line /+1 are expressed respectively by 
«/+ij=a/+ij/7/+i,. and ei+\j=q 'jfii+x^, where the ai+\j represent the observed proportions of 
modaUties of T for the line The local probability distribution q 'j,q *2,..»q y of the 
modalities of the target attribute may be expressed by: 

, aijm. + at + 1, jm + 1,. 

q J ; (2) 

Hi. + m + 1,. 

According to the ChiMerge method, we calculate the value of x^ for the lines / and 
z+l, namely, taking account of the fact that ^q'j = ^aij = l: 
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( J 2 \ 



+ m + 1„ 



/ J ^ 2 \ 



(3) 



which further gives after transformation: 



2 mm + 1,. ^ iaij - m + 1. jf 

1,1 + 1 = 2^- — 

+ + 1,. q'j 



(4) 



is a random variable following a law of with J-\ degrees of freedom. The 
ChiMerge method proposes to merge the lines / and z+1 if: 

prob[x\i^\,J -'\)<Vrob{a,K)^ pn (5) 

where prob(a,K) designates the probability that '^>ol for the law of with K degrees 
of freedom and prh is a predetermined threshold value parametrizing the method. In practice, 
the value prob(a,K) is obtained from a standard table of giving the value of a as a function 

of/?ro6(^a,Ar^ and AT. 

Condition (5) expresses that the probability of independence of S and T in terms of 
the two lines considered is less than a threshold value. The merge of consecutive lines is 
iterated as long as condition (5) is verified. The merge of two lines leads to the regrouping of 
their modalities and the summation of their effectives. For example, in the case of a numeric 
attribute with continuous values we have before merge: 







«/+l,2 
















ni+ij 










Table 2 






And after merge: 














«(+l^+«i+K2 









Table 3 

Li the patent document FR-A-2 825 168 a method is proposed that is a perfecting of 
the method that has just been described, in particular in that it makes it possible to become 
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free of the problem, in the ChiMerge method, of the choice of the parameter prh, which must 
not be too high for fear of merging all lines, nor too low for fear of not merging any pair. 

Let us suppose the case of a mono-dimensional numeric attribute S with continuous 
values. After having ordered the modalities of S, the set of these modalities can be cut up 
into elementary intervals Sf=[si^Si^\{, i=l,,,J. We wish to evaluate the degree of independence 
of this attribute with a target attribute T of modalities 7),7=1,..,J. The contingence table can 
be represented: 



S/T 


Tj 


T2 




Tj 


Total 


5i 




"1,2 




n\j 


ni.. 


A 


A 


A 


A 


A 


A 


Si 










"'V 


Si+\ 








rii+xj 




A 


A 


A 


A 


A 


A 


Si 




«/.2 




Article IV. 


«/,. 


Total 


n.i 


n.,2 




n.j 


N 



Table 4 

According to (1), the value of over the set of the table can be expressed by: 

{riij ~ eijf 



1=1 J= 



eu 



(6) 



Also, noting qi,q2,",gj, the probability distribution of the modalities of the target 

attribute, and a/,, the proportions of effectives observed for the line /, and observing that 

J J 

^if^qjrii,., nij=aijni,, and ^qj = = 1 : 



y=i 



>i 



1=1 



-1 



1 V 1j 



(7) 



1 = 1 



where x^(o is the value of for the line z. The expression (7) signifies that is 
additive with respect to the lines of the table. 

After merge of two consecutive lines / and z+l, the value of x^ is modified and the 
new value, stated as xV./+i> therefore be written: 
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xV.i+u=x'Wr?./+u (10) 

where Ax^(ij+\) is the variation of resuUing from the merge of the Unes / and i+l. It 
has been shown that the value of Ax^^/ may be calculated explicitly as a function of the 
proportions of effectives of the lines i and 

Ax\u^n = -( ^'--^^'-O y C^^-^'-'-^y (11) 

The list of the values of Ax^(ij+\) is sorted by decreasing values. For the one 
presenting the highest value, we test the following inequality of the probabilities of 
independence of S and T before merge and after merge. We test then if: 

prob(z'fiiojo . ,),(/ - 2)iJ ~ 1)) < prob(x\{I - - 1)) (12) 

If condition (12) is verified, we merge the lines io and ig+l. On the other hand, if 
condition (12) is not verified, then it is not verified for any index / in consequence of the 
decrease of prob(a,K) as a function of a. The merge process is then stopped. 

If the lines io and I'^+l have been merged, the list of values Ax^r* >/+U is updated. It is to 
be noted that this update in fact concerns only the values relative to the lines contiguous to 
the lines merged, namely the lines of indices z>l and io+2 before merge (if they exist). The 
merge process is iterated as long as condition (12) is satisfied. 

The method that is described in document FR-A-2 825 168 leads to an ad hoc 
discretization of the domain of the modalities, i.e., to a discretization that minimizes the 
independence between the source attribute and the target attribute over the set of the domain. 
As a matter of fact, this discretization method makes it possible to regroup adjacent intervals 
having similar prediction behaviors with respect to the target attribute, the regrouping being 
stopped when it harms the quality of prediction, in other words when it no longer decreases 
the probability of independence of the attributes. 

By successive merges we obtain a contingence table, the niunber of lines of which is 
reduced, and the effectives per box is increased. 
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This method nevertheless poses the problem due to a phenomenon referred to as 
"over-leaming", by which we unduly draw the conclusion of a dependence of the attributes. 
That corresponds to an improper generalization of characteristics present in the sample 
studied solely on account of statistical fluctuations. Still in the document FR-A-2 825 168, it 
was proposed, in order to resolve this problem, to adapt the discretization method described 
above in the following way: priority is first granted to the merges of lines verifying (12), 
which makes it possible to verify a minimum effective criterion. The minimum effective 
criterion can, for example, be written for the line 

eioj > \og2{\QN)J = l,..,y (13) 

Nevertheless, in spite of the good experimental results obtained, it has tumed out that 
in some cases the minimum effective criterion used above did not offer a sufficient guarantee. 
In particular, the discretization of independent attributes of the target attribute leads to a 
discretization into several intervals. That translates into an over-leaming, all the more 
important the higher the size of the leaming sample. 

Therefore the method that is set forth in the patent document FR-A-2 825 168 does 
not make it possible to define a "floor" level of the number of intervals corresponding to the 
independent attributes of the target attribute. The empirical choice of the minimum effective 
is therefore not satisfactory in the presence of attributes without predictive significance. 
Moreover, it does not take account of the number and distribution of the target modalities. 

Although the preceding introduction relates to a method of discretization of a numeric 
source attribute, this invention is not limited to such a method. As a matter of fact, the 
problem that this invention seeks to resolve, which is the problem of "over-leaming" 
mentioned above, is altogether general and also relates to methods of grouping of the 
modalities of a source attribute when said modalities are not continuous but rather discrete. 
When the modalities are continuous, they can be partitioned into elementary intervals 
whereas when they are discrete, they are partitioned into groups. It also relates to methods of 
discretization or grouping of a source attributes group, for example of the number k, which 
can then be considered as methods of discretization or grouping in dimension k. Intervals 
and groups can therefore be of dimension k. In this description, they will subsequently be 
referred to in a general way as "regions". 
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Moreover, although this introduction or the rest of the description considers as merge 
criterion the criterion (essentially for convenience of description), it is to be understood 
that this invention is not limited to this particular criterion. 

The object of this invention is therefore to propose a perfecting of a method of 
discretization / grouping of a source attribute or a source attributes group of a database 
containing a population of individuals with the object in particular of predicting modalities of 
a given target attribute, which will make it possible to prevent the phenomenon of "over- 
leaming" mentioned above from preventing the detection of attributes without predictive 
significance. 

With this end in view, and in the altogether general case, this invention relates to a 
method of discretization / grouping of a source attribute or a source attributes group of a 
database containing a population of individuals with the object, in particular, of predicting 
modalities of a given target attribute, said method comprising the following steps of: 

a) Partition of said modalities of said source attribute or said attribute group into 
elementary regions, 

b) Evaluation of a merge criterion for each pair of elementary regions, 

c) Search, among the set of all pairs of elementary regions that can be merged, for the 
pair of elementary regions for which said merge criterion would be optimized, 

e) Stopping of the method if there are no elementary regions the merge of which 
would have the consequence of improving said merge criterion, 

f) otherwise merge and reiteration of steps b) to e). 

With a view to resolving the problem mentioned above, this method is characterized 
in that it comprises in addition a step d) between steps c) and e) that skips directly to step f) 
as long as the value of a valuation variable of the merge under consideration, said valuation 
variable characterizing the behavior of said merge criterion, is not included in a 
predetermined zone of atypical values. 

According to another characteristic of this invention, said predetermined zone of 
atypical values is such that for a target attribute independent of said source attribute or said 
source attributes group, the value of said merge variable is not included in said zone with a 
predetermined probability p. 
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This invention also relates in particular to a method of discretization of a source 
attribute of a database containing a population of individuals with the object in particular of 
predicting modalities of a given target attribute, said method comprising the following steps 
of: 

a) Partition of said modalities of the source attribute into adjacent two-by-two 
elementary intervals, 

b) Evaluation for each pair of adjacent elementary intervals of said set, of the value of 
of the contingence table after a possible merge of said pair, 

c) Search, among the set of pairs of elementary intervals that can be merged, of the 
pair of elementary intervals the merge of which would maximize the value of x^, 

e) Stopping of the method if there are no elementary intervals that make it possible 
to reduce the probability of independence, 

f) otherwise merge and reiteration of steps b) to e). 

According to a characteristic of this method, it comprises in addition a step d) 
between steps c) and e) that skips directly to step f) as long as the value Ax^ of the variation 
of the value of x^ before and after merge is, in absolute value, less than a predetermined 
threshold value MaxAx^. 

According to another characteristic of the invention, said predetermined threshold 
value MaxAx^ is such that for a target attribute independent of the source attribute the value 
Ax^ of the variation of the value of before and after merge is always less than said value 
MaxAx^ with a predetermined probability p. 

According to another characteristic of the invention, said predetermined threshold 
value MaxAx^ is equal to the function of of degree of freedom equal to the number J of 
modalities of the target attribute minus one for a probability p to the power 1/N where N is 
the size of the sample of the part of the database to which said discretization method is 
applied: 

where Invy^ is the function that gives the value of x^ as a function of a given 
probability p. 
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According to another characteristic of the invention, said method comprises a step for 
verification that the effective of a source attribute for modaUties in a given interval for each 
target attribute is greater than a predetermined value, and if such is not the case, to implement 
the merge of said interval w^ith an adjacent interval. 

This invention also relates in particular to a method of grouping of a source attribute 
of a database containing a population of individuals with the object in particular of predicting 
modalities of a given target attribute, said method comprising the following steps of: 

a) Partition of said modalities of the source attribute into a plxirality of 
groups, 

b) E valuation for each pair of groups of said set, of the value of of the contingence 
table after a possible merge of said pair, 

c) Search, among the set of pairs of groups that can be merged, for the pair of groups 
the merge of which would maximize the value of x^, 

e) Stopping of the method if there are no merges of groups that make it possible to 
reduce the probability of independence, 

f) otherwise merge and reiteration of steps b) to e). 

According to a characteristic of the invention, this method comprises in addition a 
step d) between steps c) and e) that skips directly to step f) as long as the value Ax^ of the 
variation of the value of before and after merge is, in absolute value, less than a 
predetermined threshold value MaxAx^. 

According to another characteristic of the invention, said predetermined threshold 
value MaxAx^ is such that for a target attribute independent of the source attribute the value 
Ax^ of the variation of the value of x^ before and after merge is always less than said value 
MaxAx^ with a predetermined probability p. 

According to another characteristic of the invention, in order to establish the 
predetermined threshold value MaxAx^, it consists in using a previously calculated table of 
values of mean and standard deviation as a function of the number of modalities of the source 
attribute and of the number of modalities of the target attributes, to determine by linear 
interpolation.from said table of values the mean and standard deviation of MaxAx^ 
corresponding to the attributes to be grouped, and then to determine by using the inverse 



1 73943 Ivl 



-46- 



normal law the corresponding predetermined threshold value MaxAx^, which will not be with 
a probability p. 

According to another characteristic of the invention, for two target modalities, the 
mean of MaxAx^ is asymptotically proportional to 2//;r where I is the number of source 
modalities. 

According to another characteristic of the invention, for two source modalities, the 
law of MaxAx^ is the law of with J-1 degrees of freedom, J being the number of target 
modalities. 

According to another characteristic of the invention, said method comprises a prior 
step of verification that the effective of a source attribute for modalities in a given group for 
each target attribute is greater than a predetermined value, and if such is not the case, to 
implement a merge of said group with a specific group, said merged group then forming 
again said specific group. 

This invention also relates in particular to a method of discretization in dimension k of 
a group of k continuous source attributes of a database containing a population of individuals, 
with the object in particular of predicting the modalities of a given target attribute, said 
method comprising the following steps of: 

a) Partition of said modalities of the group of k source attributes into elementary 
regions of dimension k, 

b) Evaluation for each pair of adjacent elementary regions, of the value of of the 
contingence table after a possible merge of said pair, 

c) Search, among the set of pairs of regions that can be merged, for the pair of regions 
the merge of which would maximize the value of x^, 

e) Stopping of the method if there is no set of intervals that make it possible to reduce 
the probability of independence, 

f) otherwise merge and reiteration of steps b) to e). 

It is characterized in that it comprises in addition a step d) between steps c) and e) that 
skips directly to step f) as long as the value Ax^ of the variation of the value of before and 
after merge is, in absolute value, less than a predetermined threshold value MaxAx^. 
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Finally, it relates to a method of grouping in dimension k of a group of k discrete 
source attributes of a database containing a population of individuals, with the object in 
particular of predicting the modalities of a given target attribute, said method comprising the 
following steps of: 

a) Partition of said modalities of the group of k source attributes into a plurality of 

groups, 

b) Evaluation for each pair of groups of the value of of the contingence table after a 
possible merge of said pair, 

c) Search, among the set of pairs of groups that can be merged, for the pair of groups 
the merge of which would maximize the value of x^, 

e) Stopping of the method if there are no merges of groups that make it possible to 
reduce the probability of independence, 

f) otherwise reiteration of steps b) to e). 

It is then characterized in that it comprises in addition a step d) between steps c) and 
e) that skips directly to step f) as long as the value Ax^ of the variation of the value of 
before and after merge is, in absolute value, less than a predetermined threshold value 
MaxAx^. 

The characteristics of the invention mentioned above, as well as others, will appear 
more clearly upon reading of the following description of an example of realization, said 
description being done with relation to Fig. unique is a flowchart showing the various steps 
implemented by the method of discretization or a method of grouping according to this 
invention. 

As already mentioned above, this description will, for reasons of convenience, 
consider as: 

merge criterion, the x^ criterion, 

improvement of the merge criterion, the reduction of the probability of independence, 

valuation variable of a merge, the value of the variation Ax^ of the value of x^ before 
and after said merge, 

zone of atypical values, the values of the variation Ax^ greater than a predetermined 
threshold value MaxAx^. 
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But it is to be understood that this invention is not hmited to these particular cases. 

At first, we will consider, in this limiting context set forth above, a method of 
discretization of a source attribute such as the one that is described in the patent document 
FR-A-2 825 168. In this document, we consider all possible merges of intervals, we choose 
the best merge, and if the stopping criterion is not attained, we carry out this merge and 
continue. 

According to this mode of realization of this invention, we will in the same way study 
the law of Ax^/,/+i (variation of the value of at the time of the merge of two intervals / and 
/+!). At the time of the unfolding of the method a large number of merges are considered, 
and at each step we choose the best of all these merges by optimizing the ^ criterion, or, 
which is equivalent, by optimizing the Ax^ criterion (the starting x^ being fixed) in a way 
equivalent to that described in the document mentioned above. In addition to a stopping 
condition on the probabilities of independence between source attribute and target attribute 
before and after, the method according to this invention provides for the continuation of the 
merges as long as the value of AxV/o+/ is not sufficiently large (It is to be recalled here that 
iO and iO+1, respectively, are the indices of the intervals whose value of Ax^/o./o+y is the 
highest). 

In other words, we will carry out a test on this highest value of /Sriiojo-^i^ or more 
exactly its absolute value, by comparing it with a maximal value designated MaxAx^. If this 
absolute value of Ax^iojo+i is less than the value MaxAx^, then the process of merge of the 
intervals is forced no matter what (not knowing the other stopping conditions). 

A flowchart of an example of implementation of a method of discretization according 
to this invention is represented in Fig. 1. 

The algorithm begins with an initialization phase 100, 110, 120, 130 (the references 
are identical to those used in the patent document FR-A-2 825 168 wherein we carry out a 
partition of the domain of the modalities of the source attribute into ordered elementary 
intervals (step 100), we calculate the value of the resultant x^ as well as the values for the 
/lines of the contingence table (step 1 10), we calculate the values Ax^aj^jj of the values x^(i) 
(step 120) and we sort these values Ax^(u+j) by decreasing values (step 130). 

It is to be noted that the first value Ax^iojo+i is the one that is the highest in relative 
value, but as the values Ax^(u+j) are always negative, it is the one whose absolute value is the 
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lowest. This value corresponds to the merge of two adjacent intervals with indices iO and 
iO+1 for which the absolute value of Ax^iojo+i is minimized or for which the value of X^foojo+i) 
after merge of the intervals iO and iO+1 is maximized. 

In step 200, a step that is new with respect to what is described in document FR-A-2 
825 168, we initialize the value MaxAx^. It could be a matter of a constant value taken once 
and for all. Nevertheless, as we will see later on, this value depends on the data to be treated 
so that at step 200, it is a calculation that is carried out. 

In step 140, we test whether the minimum effective condition in each cell of the 
contingence table is verified. It may be a matter of verifying that each cell of the table 
comprises an effective minimum in order that the process of this invention may fiinction 
correctly while being placed under the application conditions of the test. It is to be 
understood that it is not a question here, as was the case in the patent document FR-A-2 825 
168 mentioned above, of resolving the problem of over-leaming. Again employing the 
notations above, it is a matter here of verifying that: 

rimin for all i and j 

where rimm is the minimum effective number. This number is, for example, 5. 

In the case in which the preceding relation is verified, we pass directly to test 210. In 
the negative, we proceed by step 145. 

In step 145, we give priority to the pairs of intervals for which at least one among 
them has a cell that hasn't attained the minimum effective nmin and in step 165 we select 
among them the pair of intervals (zo,zo+l) for which the value Ax^wjo+i is the highest. We 
then proceed to step 170. 

In step 210, a step that is new with respect to what was described in document FR-A- 
2 825 168, we test whether the highest absolute value ofAx^iojo+i is less than the maximal 
value designated MaxAx^ determined in step 200. If this absolute value of Ay^io,io^\ is less 
than the value MaxAx^, we then proceed to step 160, otherwise we go to step 150. 

In step 150, we consider the intervals iO and iO+1 for which the value Ax^io.i(?+i is the 
highest and we test whether the probability of independence between source attribute and 
target attribute after merge of these two intervals, designated prob[x^f(iojo + o, (/ - 2){J - 1)), 
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is less than or equal to the probability of independence between source attribute and target 
attribute before merge of the two intervals. We therefore test the following relation: 

prob[x'nio, io . 1), (/ - 2)(y - 1)) < prob(z' , (/ - l)iJ - 1)) 

If such is the case, we select (step 160) the pair of intervals iO and iO+las being to be 
merged and we proceed to step 170. On the other hand, if such is not the case, the process is 
ended at 190. 

In step 170, the intervals of index io and /o+l are merged. The new value of ^^(/o) is 
then calculated in 180 as well as the new values of ^X^o^ - and Ax^oojo + o for the 
adjacent intervals, if they exist. In 185, the list of the values Az^OJ + o is updated: the old 
values Ax^iio - i,io) and A;if^(/o,/o + o are deleted and the new values are stored. The list of the 
values Ax^iu + o is advantageously organized in the form of a binary tree of balanced search 
that makes it possible to manage the insertions / deletions while maintaining the relation of 
order in the list. Thus it is not necessary to completely sort the list at each step. The list of 
flags is also updated. After the update, the process returns to the test step 140. 

We describe below modes of realization of means that make it possible to determine 
the value of MaxA/,^. It is to be understood that these means sure implemented in the box 200 
of Fig. 1. 

In order to do this, we will start from the observation that, for a source attribute and a 
target attribute that are independent, the desired result is that at the conclusion of the process 
of discretization, only a single interval remains any longer, signifying in this way that the 
source attribute (taken separately) does not contain any information on the target attribute. In 
this case, we can for a given probability p determine a value MaxAx^(p) that will not be 
exceeded with a probability p. 

Thus, in step 200, we determine MaxAx^ as being equal to MaxA5c^(p), with p a 
probability whose value is predetermined. 

In this way we ensure in this way the desired behavior with a probability p. In the case 
of any two attributes (not necessarily independent), this way of making the method reliable 
makes it possible for us to assert that if the algorithm produces a discretization containing 
information (at least two intervals), there is a probability greater than p that the descriptive 
attribute is really the carrier of information about the attribute to be predicted. 
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We sought to theoretically determine the relation that exists between the value of 
MaxAx^ and the probability p. In order to do this, we studied the law of Delta Ax^^i.i+u 
(variation of the value of at the time of the merge of two intervals of rank i and i+1) in the 
case of two independent attributes. In this case, it is necessary to continue the merges until 
there no longer remains but a single final group, which is in fact the initial sample. It is 
therefore necessary that the largest value A2'^(»o,/o + o encountered during the process be 
accepted. We will try to estimate this largest value during the unfolding of the discretization 
process, and impose that the merges be continued as long as this threshold is not attained, 
which will therefore be the sought- for value of MaxA/,^. 

For two independent attributes, the value of follows a law of probability whose 
expectation and variance are linked in the following way: 

We have also been able to show (see previously, relation 11) that the induced 
variation of following the merge of two intervals of respective effectives n and n' and of 
proportions of target local modalities respectively equal to pj and p'j can be written in the 
form: 




Pj is the global proportion of modalities of the target attribute of rank j. 

It is known that this variation is always negative, and is zero only if the intervals are 
identical or have exactly the same proportions of target modalities. Thus, it is known that 
of a contingence table can only decrease following the merge of two lines of the contingence 
table. Afterwards, we redefine Ax^ by its absolute value in order to manipulate only positive 
magnitudes. 

n + n^j:l Pj 
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The calculation of the distribution function of Ax^ is based on discrete binomial laws, 
which makes it difficult to evaluate for large values of n. We will use the central limit 
theorem to approximate the law of Ax^ in the case where n=n\ 

We make the following proposition: for a source attribute independent of a target 
attribute with J modalities, Ax^ resulting from the merge of two intervals of the same 
effective n and n' asymptotically follows a law of x^ with J-1 degrees of freedom. 

We have been able to show that this proposition is not only valid in the case of two 
target modalities but also in other cases. 

We observe that the law of Ax^ depends on the number of modalities of the target 
attribute, but not on their distribution. 

We will now evaluate the statistics of the merges of the method according to this 
invention. 

We observe first that at the time of a "total" discretization up to a single final interval, 
the number of merges carried out is approximately equal to the size N of the sample. 

We will at first experimentally evaluate the real behavior of the algorithm and thus 
this simple statistical modeling of the method of this invention. The experimentation consists 
in implementing the method of the invention on a sample comprising a continuous source 
attribute independent of the target attribute and taking equi-distributed Boolean values. We 
carry out all possible merges up to the point of obtaining a unique terminal interval (the 
stopping criteria are made inactive) and we collect the value of Ax^ of each of these merges in 
order to plot the distribution fimction from them. We carry out this experimentation on 
samples of size 100, 1,000 and 10,000, and then we compare the distribution fimctions 
obtained with the theoretical distribution fimction of Ax^ of two intervals of the same 
effectives (law of x^ with one degree of freedom). 

This experimentation shows that the law of the Ax^*s resulting from the various 
merges carried out at the time of the implementation of the method of the invention does not 
depend on the size of the sample, and is well modeled by the theoretical law of Ax^ 
demonstrated above for two intervals of the same effective. According to a mode of 
realization of this invention, a threshold MaxAx^ for the implementation of the above method 
is such that for two independent source and target attributes, the method converges toward a 
single terminal group with a probability greater than p (p=0.95 for example). It is therefore 
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necessary that all merges considered be accepted, i.e., that all the values of A^^ resulting from 
the merges considered be less than the threshold MaxAx,^. By being based on the preceding 
modeling wherein all merges are independent, the probability that all merges considered are 
accepted is equal to the probability that one merge is accepted to the power N. We therefore 
seek MaxAx^ such that: 

P[/^X^J < Maxj^x^y > p 

Proceeding by the equivalent law of x^, we have: 
p[x^j-\<Maxls.x^)> p'^ 
Which can also be written: 
Max/^x^ = Invx^J-{p'^ 

where Inv;^ is the function which gives the value of as a function of a given 
probability p. 

We sought to validate this modeling of the law of MaxAx^. hi order to do so, we were 
interested this time not in the distribution of the values of Ax^ during the implementation of 
the method of the invention, but in the maxima of these values. For that, we use samples of 
two really independent source and target attributes as previously and we collect, for a large 
number of samples for discretization, the maximal value of the Ax^'s resulting from the 
merges of intervals effected. We carry out this experimentation 1000 times for samples of 
size 100, 1,000 and 10,000 and 100,000 and we plot the "empirical" distribution functions of 
MaxAx^ for each of these interval sizes. We also plot the theoretical distribution functions 
obtained with the above formula on the same figures. 

We observed that the empirical laws and the corresponding theoretical laws have very 
similar forms, whatever the size of the sample. We also observed that the theoretical values 
constitute an upper limit of the empirical values. Consequently, this limit constitutes a 
sufficiently faithful estimation of the empirical values. It is to be noted that although resting 
on reasonable bases, its behavior as upper limit could be verified only experimentally. 

We carried out experimentations that make it possible to evaluate this invention in its 
first particular mode of realization. 
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In a first experimentation, we discretized a continuous source attribute independent of 
a target attribute to be predicted, for sample sizes of 100, 1,000, 10,000, 100,000 and 100,000 
[sic]. For each sample size, we repeated this experimentation 1,000 times. We coxmt the 
number of cases in which the discretization leads to a unique terminal interval, and in the 
contrary cases of multi-interval discretization, we calculate the mean value of the number of 
intervals. The results of this first experimentation are shown in the table below. 







Multi-interval 
discretization 


Sample size 


% without 
discretization 


Number of intervals 


100 


98.6% 


2.36 


1,000 


98.7% 


3.00 


10,000 


98.4% 


3.00 


100,000 


97.2% 


3.00 


1,000,000 


95.6% 


3.00 



It can be noted that the discretization of an attribute independent of the target attribute 
leads in 95% to 98% of the cases to a unique terminal interval. It can be concluded, on the 
basis of this experimentation, that the method according to this invention behaves in a way in 
keeping with what is expected, at least in the domain of sample sizes varying firom 100 to 
1,000,000. 

We will show below that the method that has just been described in relation to Fig. 1 
is not only applicable to the problem of discretization of numeric data as shown above but 
also to the problem of grouping of the modalities of symbolic attributes. 

It is to be recalled that the problem of the grouping of the modalities of a symbolic 
attribute consists in partitioning the set of values of the attribute into a finite number of 
groups, each identified by a code. Thus, most of the predictive models based on a decision 
tree use a grouping method to treat symbolic attributes, in such a way as to combat 
fi'agmentation of the data. 
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The management of the modalities of a symbolic variable is a more general problem 
the stakes of which amply exceed the bounds of decision trees. For example, the methods 
based on neuron networks using only numeric data often resort to a complete disjunctive 
coding of the symbolic variables. Li the case in which the modalities are too numerous, it is 
necessary, as a preliminary, to conduct groupings of modalities. This problem is also 
encountered in the case of Bayesian networks. 

At stake in the regrouping of modalities is the finding of a partition realizing a 
compromise between informational quality (groups homogeneous with respect to the source 
attribute to be predicted) and statistical quality (sufficient effectives to ensure an effective 
generalization). Thus, the extreme case of an attribute having as many modalities as 
individuals is unusable: any regrouping of the modalities corresponds to a learning "by 
heart" that is unusable in generalization. In the other extreme case of an attribute possessing 
a single modality, the capacity for generalization is optimal, but the attribute does not possess 
any information that would make it possible to separate the classes to be predicted. It is then 
a matter of finding a mathematical criterion that makes it possible to evaluate and compare 
partitions of different sizes, and an algorithm that leads to finding the best partition. 

The grouping method according to this invention uses the global value of of the 
table of contingence between discretized attribute (source attribute) and attribute to be 
predicted (target attribute), and seeks to minimize the corresponding probability of 
independence P. The grouping method begins with the partitioning of the initial modalities 
and then evaluates all possible merges and finally chooses the one that maximizes the 
criterion of applied to the new partition that was formed. The method stops automatically 
as soon as the probability of independence P no longer decreases. This part of the method is 
identical to the one that is described in document FR-A-2 825 168. Moreover, the grouping 
method according to this invention is similar to the discretization method described above 
while bringing to it the same perfection. It makes possible a real control of the predictive 
quality of a grouping of modalities. 

Like the discretization method described above, it rests on the study of the statistical 
behavior of the algorithm in the presence of a symbolic attribute independent of the attribute 
to be predicted. We therefore studied the statistics of the maximal variation of the x^criterion 
at the time of the complete unfolding of the grouping algorithm. This study showed that this 
maximal value MaxAx^ depends only on the number of modalities of the source and target 
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attributes and is insensitive to the distribution of these modaUties as well as to the size of the 
learning sample. With reference to the modeling of the statistics of MaxAx^, we then 
modified the initial grouping algorithm by constraining it to accept any merge of modalities 
that leads to a variation of less than the calculated maximal theoretical variation MaxAx^. 

This invention makes it possible to guarantee, on the one hand, that the modality 
groupings of an attribute independent of the attribute to be predicted leads to a single terminal 
group and, on the other hand, that the groupings leading to several groups correspond to 
attributes having a real predictive significance. Experimentations confum the significance of 
this robust version of the algorithm and show good predictive performances for the groupings 
obtained. 

The discretization method described previously can be generalized to grouping by 
replacing the intervals by groups of modalities and by replacing the search for the best merge 
of adjacent intervals by the search for the best merge of any groups. 

The minimum effective constraint is expressed here by a minimum effective per 
modality. At the time of a pre-treatment, any source modality not attaining this minimum 
effective will be unconditionally grouped in another special modality provided for this 
purpose. Thus, there remain then only modalities that satisfy the minimum effective 
constraint entering into the grouping method. 

Li a manner analogous to the discretization method previously described, it is possible 
to reduce the grouping algorithm to an algorithmic complexity of Nlog(N)+J^log(J) where N 
is the number of individuals in the sample and J is the number of modalities of the source 
attribute (once the other special modality is treated). 

The flowchart of the grouping method according to this invention is identical to that 
of the discretization method described above in relation to Fig. 2. 

We will now seek to express the value of MaxAx^ in the context of a grouping 
method. 

At the time of the implementation of the grouping method according to the invention 
as illustrated in Fig. 2, we consider all possible merges of lines of the contingence table and 
we choose the one that maximizes the value of the contingence table after merge of the 
lines, i.e., the one that maximizes the Ax^ variation during the merge. 
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We consider that the value MaxAx^ is the maximal value of Ax^ that will be attained 
at the time of the implementation of the method according to this invention, the value 
obtained at the time of the attainment of a unique terminal group of modalities. 

Thus, the basic principle of the method of this invention is to establish that for a 
source attribute independent of the attribute to be predicted, we will naturally observe 
variations of A^^ and therefore a MaxAx^ due to the chance of the sample. But in short, the 
grouping of the modalities of an attribute independent of the attribute to be predicted should 
lead to a single terminal group. Consequently, we impose that any group merge leading to a 

variation less than the variations that can be due to chance (i.e., less than MaxAx^) is 
automatically accepted. In this way we also ensure that any grouping leading to at least two 
terminal groups corresponds to an attribute not independent of the attribute to be predicted. 

We will now seek to establish the statistics of MaxAx^ in the case of the treatment of 
the grouping of modalities of attributes. 

Let N be the size of the sample, I the number of source modalities and J the number of 
target modalities. 

It is to be noted that, for reasons already explained above, we consider the case 
wherein the minimum effective constraint of 5 per cell of the contingence table is respected, 
in such a way as to be able to validly use the x^ statistics. 

A priori, the MaxAx^ statistics depend on the size of the sample N, on the number of 
modalities of the source attribute I, on the number of modalities of the attribute J, but also on 
the distribution of the frequencies of the source modalities and on the distribution of the 
frequencies of the target modalities. 

In fact, we demonstrated that the MaxAx^ law depends in reality only on the number 
of modalities of the source attribute I and of the target attribute J. We also demonstrated that 
for 2 source modalities, the MaxAx^ law is the law of with J-1 degrees of freedom. Its 
mean is therefore J-1. 

Moreover, for 2 target modalities, we also demonstrated that the mean of MaxAx^ is 
asymptotically proportional to 21/ 7t. 

We have described up to now a method of discretization of a source attribute whose 
continuous modalities are mono-dimensional but it is to be understood that this invention is 
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also applicable to a method of discretization of a source attribute whose equally continuous 
modalities are of dimensions k. 

hi this case, the source attribute is a numeric source attribute of dimensions k formed 
by k mono-dimensional source attributes. Each individual of the population may be 
represented by a point of the space of said attributes of dimension k. 

This method of discretization in dimension k of a group of k source attributes 
therefore consists in doing a partition of the modalities of the group of the k source attributes 
into elementary regions of dimension k and an evaluation for each pair of adjacent 
elementary regions of the value of of the contingence table after a possible merge of said 
pair. 

It is to be noted that the elementary regions in question are, for example, Voronoi 
cells of the space of the source attributes. In order to find two adjacent elementary regions, 
we construct the Delaunay graph associated with the Voronoi' cells and we eliminate from 
this graph any arc joining two neighboring cells by passing through a third, the pairs of 
adjacent regions being given by the arcs of the Delaunay graph after the elimination step. 

Patent document FR-A-2 825 168 can profitably be referred to for details concerning 
these steps of partition and evaluation. 

Next we carry out the merge, among the set of pairs of regions that can be merged, of 
the pair of regions the merge of which maximizes the value of and we stop the method 
when there is no set of intervals that make it possible to reduce the probability of 
independence. If such is not the case, we reiterate the preceding steps. 

According to a characteristic of this invention, the method of discretization in 
dimension k of a group of k source attributes is characterized in that it comprises in addition a 
step that skips directly from the merge step after the stopping step as long as the value Ax^ of 
the variation of the value of before and after merge is, in absolute value, less than a 
predetermined threshold value MaxA^^. 

In the same way, the method which has just been described is also applicable to the 
grouping in dimension k of a group of k discrete source attributes. As previously, it then 
consists in doing a partition of said modalities of the group of k source attributes into a 
plurality of groups and an evaluation for each pair of groups of the value of of the 
contingence table after a possible merge of said pair. 
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It consists in doing the merge, among the set of pairs of groups that can be merged, of 
the pair of groups the merge of which maximizes the value of and in stopping the method 
if there are no merges of groups that make it possible to reduce the probability of 
independence, otherwise we reiterate the preceding steps. 

This grouping method comprises in addition a step that skips directly to the reiteration 
step as long as the value Ax^ of the variation of the value of before and after merge is, in 
absolute value, less than a predetermined threshold value MaxAx^. 

It is to be recalled that in an altogether general way, this invention relates to a method 
of discretization / grouping of a source attribute or of a source attributes group of a database 
containing a population of individuals with the object in particular of predicting modalities of 
a given target attribute. 

If we refer to Fig. unique, the steps of partition of said modalities of said source 
attribute or of said attribute group into elementary regions, of evaluation for each pair of 
elementary regions of the value, after a possible merge of said pair, of a merge criterion, and 
of search, among the set of pairs of elementary regions that can be merged, for the pair of 
elementary regions for which the merge criterion would be optimized corresponding to steps 
100, 110, 120 and 130. 

The stopping step of the method if there are no elementary regions whose merge 
would have the consequence of improving the merge criterion is step 150. 

The merge and reiteration step is represented by the loop including 160, 170, 180 and 

185. 

The step that skips directly as long as the value of the valuation variable of the merge 
is not included in a predetermined zone of atypical values is step 210. 

Finally, the determination step of the predetermined zone of atypical values is step 

200. 
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ABSTRACT 

— Method A method of discretization / grouping of a source attribute or of a source 
attributes group of a database . This invention r e lat e s to a m e thod of discr e tization / grouping 
of a sourc e attribut e or of a sourc e attributes group of a databaso containing a population of 
individuals with the object in particular of predicting modalities of a given target attribute. 
The method includes the steps of: 

This invention r e lat e s to a m e thod of discretization / grouping of a sourc e attribut e or 
of a sourc e attributes group of a databas e containing a population of individuals with the 
obj e ct in particular of predicting modalities of a given target attribute. The method includes 
th e steps of: 

a) partitioning of the modalities of the source attribute or the attributes group into 
elementarv regions, 

a) Partition of said modalities of said source attribut e or said attribute group into b} 
evaluating of a merge criterion for each pair of elementary regions, 

b) Evaluation of a merg e crit e rion for each c) searching, among the set of pairs of 
elementarv regions that can be merged, for the pair of elementary regions for which the 
merge criterion would be optimized, 

c) S e arch, among the s e t of pairs of elementary r e gions that can bo m e rg e d, for th e pair 
of e lementary regions for which the m e rge crit e rion would bo optimiz e d, 

d) skipping directlv to step f) as long as the value of a valuation variable of the merge 
under consideration is not within a predetermined zone of atvpical values, 

^) [[S]] stopping of the method if there are no elementary regions whose merge would 
have [[the]] a consequence of improving said merge criterion, and 

f) otherwise m e rge merging and reiteration of steps b) to e). 

According to th e inv e ntion, it comprises in addition a step d) betwe e n st e ps c) and o) 
that sldps dir e ctly to st e p f) as long as th e valu e of a valuation variable of th e m e rg e und e r 
consideration is not comprised in a pr e d e termined zone of atypical valu e s. 

Fig. unique 
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